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ABSTRACT 


The use of Redundant Asynchronous Multiprocessor System to achieve Ultra reli- 
able Fault Tolerant Control Systems shows great promise. The development has been 
hampered by the inability to determine whether differences in the outputs of redundant 
CPU’s are due to failures or to accrued error built up by slight differences in CPU clock 
intervals. This study derives an analytical dynamic model of the difference between re- 
dundant CPU’s due to differences in their clock intervals and uses this model with on-line 
parameter identification to identify the differences in the clock intervals. The ability of 
this methodology to accurately track errors due to asynchronisity is demonstrated usin^ a 
simulated multiprocessor system. The algorithms generate an error signal with the effect 
of asynchronisity removed and this signal may be used to detect and isolate actual System 
failures. 



I. BACKGROUND 


Increased reliance on computers and distributed processing in flight control systems 
requires higher reliability. A new approach to achieving higher reliability is to distribute 
sequential computational operations over multiple Microprcessor Central Processing Units 
(CPU’s.) Increased reliability of the sequential CPU’s is then achieved using parallel 
pipelines of sequential CPU’s, with each CPU comparing the results of the parallel oper- 
ations at the previous sequential stage to identify and isolate failures. Such a Redundant 
Asynchronous Multiprocessor System (RAMPS) has been designed and tested [1]. Advan- 
tages of this system include simplifled fault detection and isolation logic since the results 
at each sequential computation are generated redundantly for comparison. Also the re- 
liability is higher than other redundant architectures since the operation of each CPU .is 
independent of failures in any other CPU. 

Using totally autonomous CPU’s to achieve high reliability results in a major im- 
pediment to successful implementation of this system. Since each CPU must have its own 
clock to be totally independent of other CPU’s, the parallel computations will be performed 
asynchronously. Small differences in the CPU clocks can propagate into large differences 
in the parallel computations over a period of time and the fault detection logic will be 
unable to distinguish between errors due to CPU failures and errors due to asynchronisity. 

One approach to resolving the problem of asynchronisity heis been to use asymptoti- 
cally stable control laws in each CPU [2]. This eissumes that the error due to eisynchronisity 
will be bounded. The disadvantage of this approach is that unrealistic constraints on the 
control laws may be required to reduce the error due to asynchronisity to acceptable levels. 
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II. OBJECTIVE AND OUTLINE OF REPORT 

The objective of this study is to examine the nature of the error due to asynchronisity 
to determined if it is possible to model and track this source of error. If this is possible, 
then this information can be used by the fault detection logic to account for differences in 
parallel CPU’s due to asynchonisity and it can also be used to periodically resynchronize 
the CPU’s to prevent excessive build up of this error source. 

This report begins with a description of the approach used in accomplishing this ob- 
jective. A description of the RAMP system and the simulation of this system on a single 
CPU computer then follows. The next section defines the error due to asynchronisity and 
examines the nature of this error using the computer simulation of the RAMP system. A 
mathematical model of this error is then developed and tested against the error generated 
by the simulation. Finally, on-line parameter identification is applied, using the developed 
error model, to identify the difference in the CPU clocks and use this information to esti- 
mate the error due to asynchronisity on-line. The algorithms are teste using the simulated 
RAMP system. 


III. APPROACH 

In order to efficiently attack the problem, a simulation program was implemented to 
study the behavior of RAMPS. The approach now can be presented in the following order: 

1. Determine operational requirements to insure analytical error function: The def- 
inition of the error function affects the nature of the error. The difference in the output 
of two assynchronous CPU’s is a highly nonanalytic function, however by "synchronizing” 
the time at which this error is observed by a downstream CPU, the observed error function 
can be made smooth and hence analytically describable. 

2. Derive a mathematical model of the analytical error as function of the difference in 
CPU clock intervals:The error between parallel asynchronous CPU’s will be defined and 
the analytical error function will be derived by applying Sample Data theory. 

3. Identify the difference in CPU clocks by applying regression to error data with the 
error model as a constraint. 

4. Use the identified difference in the CPU clocks to track the error due to asynchro- 
nism. The voting logic can then be based on other sources of error. Two voting logics will 
be compared here: 


- Midvalue selection and 

- First value selection. 
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IV. ASYNCHRONOUS MULTIPROCESSOR SYSTEM DESCRIPTION 

RAMPS can be described graphically in the following diagram where boxes represent 
CPU’s and circles represent buffers. 


Sensors 

(continuous) 


CPU A’s 


CPU B’s 


CPU C’s 


#1 


■■ 
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Fig. 4.1 Schematic Diagram of RAMP system- 

The major characteristics of this system are as follows: 

- The first parallel sequence consists of sensors. These sensors give continuous outputs. 

- Each CPU is operating independently of the others. 

- Each CPU outputs at a fixed frequency to buffers which are read by the downstream 
CPU’s. 

- Parallel CPU’s at the same sequential position perform the same task. Their exe- 
cution times are slightly different (within a known tolerance and about a known nominal 
time.) 

- Sequential CPU’s in the same parallel pipeline perform different tasks. 

There are three kind of faults which can occur in the RAMPS modules[l]: 

- Faults that cause data alteration (Fig. 4.2) 

- Faults that result in improper execution of the designed code (Fig. 4.3.) 

- Faults that cause excessive timing differences in parallel CPU’s. 

The purpose of this study is to isolate and identify errors due to timing differences so 
that the other faults are not mausked by the effect of the timing differences. 
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FIC. 4.5. FAULTS RESULTING IH IMPROPER PROGRAM EXECVmOH 



V. SIMULATION PROGRAM 


A simulation of the RAMP system was implemented on a Micro- Vax II computer. 
The purpose of the simulation was to investigate the differences in the output of parallel 
CPU’s due to differences in their internal clock rates. In order to illustrate the effect of 
these clock differences the following example is given. 

- Each of two CPU’s h^ a 5 Mega hertz (Mhz) clock that is different by 0.001%. The 
clock difference (AC) is then: 


AC = —^—0.001% = 2. X 10 ^^sec./clockcycle 
5Mhz 

- If the computation cycle (T) for the CPU’s is 50 milliseconds (mS) the number of 
clock cycles that it will take to produce one computation cycle difference between the 
CPU’s is: 


ClockCycles = 


T 

AC 


50 X 10 ^sec.Jcomp.cycle. 
2 X 10~^‘^sec.fclockcycle 


= 25 X 10^ clockcycles/comp.cycle 


- The elapsed time (t) over which the CPU’s will develops a one computation cycle 
(50 mS) asynchronisity is then: 


clockcycles f comp.cycle 
clockcycle/sec. 


25 X 10® 
5 X 10® 


= 5000.sec/ comp.cycle = 83min.20sec. 


Since most flights would run in excess of one hour, this problem will be significant. In 
order to facilitate the analysis and minimize the require computer time for the analysis, 
greatly exaggerated clock differences are used in the simulation. 

In the simulation, each CPU solves an algorithm of the form: 


Where the supercript (m = A, B, C) denotes the parallel pipeline and the subcript 
(n = 1, 2, 3, ...) denotes the computation cycle number. The following table shows the 
coefficients of the algorithm used at each sequential CPU along with the computation cycle 
times for the three parallel CPU’s at each sequential node. 
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. 

CPU-A 

CPU-B 

CPU-C * 

1 1 

A1 1 A2 1 A3 
1 1 

^ 1 

Bl 1 B2 1 B3 
1 1 

Cl 1 C2 1 C3 

time 

required 

1 1 

0.02 1 0.0202 10.0201 

1 i 

0.035|0. 035210. 0351 
1 1 

0.018(0.0182(0.0181 

.1 1 

CX. 

0.88 

0.93 

0.85 

p 

0.12 

0.07 

0.15 


The structure and data flow of the simulation program is shown in Fig. 5.1. On 
each peiss the Executive flrst calls the sensor subroutine to generate input data for the A 
CPU’s. If any of the three parallel A CPU’s axe due to receive data on this pass, the CPU 
A subroutine is called and the results are stored in the output buffer of the appropriate A 
CPU. The same process is applied to the B and C CPU’s on each pass. 



Fig. 5.1 Data flow diagrEun. 

Figure 5.2 shows the internal logic of the Executive. Real time (t) is incremented 
by an amount DT chosen to be much smaller them the difference in computation cycle 
times. The next interrupt time for each of the j (j=l,2,3) CPU’s at the A, B and C 
nodes is then computed. The sensor subroutine is called each real time pass to simulate a 
continuous sensor input to CPU-A. For purpose of the simulation, the sensor outputs are 
modeled as pure sinusoids. Random noise is added to each of the three sensor outputs. 
Current real time is then compeired to the interrupt times for all CPU’s and any CPU’s 
requesting data on the current data pass are processed and the output stored in the 
appropriate buffer. Whenever a CPU is processed, an incremented computation cycle 
count (tA(i),*B(y)>tc(i)) is returned for use in computing its next interrupt time. 
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Figure 5.2: Executive Flowchart 
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VI. DEFINITION OF ERROR DUE TO ASYNCHRONISITY 

Denoting the output of CPU-Al as Al and the output of CPU-A2 as A2 then the 
error may be expressed in two ways: 

ERROR AT CPU-Al AND CPU-A2 OUTPUT TIMES 

The error, A1-A2, is computed at each output interval for either CPU. The error is 
seen to flip back and forth between two curves. One curve represents the error when CPUl 
outputs and the other curve represents the error when CPU2 outputs. The result of this 
error definition is shown in Fig. 6.1. 



Fig. 6.1 Error at CPU-Al and CPU-A2 Output Times 
ERROR AT CPU-B INPUT TIMES 

The error, A1-A2, is computed at points where CPU-B goes to the buffer to obtain 
the outputs of the A CPU’s. Since the B CPU’s have clock times that are out of synchro- 
nization with the A CPU’s, B will sometimes sample when Al has output most recently 
and sometimes when A2 has output most recently. The result is a sporadic oscillation 
between the two bounding error curves, as shown in Fig. 6.2. 


SIMHtAW 



Fig. 6.2 Error at CPU-B Input times 
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The non-analytic nature of the error as seen by CPU-B significantly complicates the 
modeling process. This problem can be eliminated by defining the error such that the most 
recent output used in the error definition always comes from the same CPU, This assures 
that the error will always stay on the same bounding curve. This synchronization process 
is shown in Fig. 6.3. In this figure, the output times associated with CPU’s Al, A2 and B 
are superimposed. The error associated with using the two most recent outputs, as seen by 
CPU B, is shown under the chart. If each CPU tags its output with a computation cycle 
number so that the error can be defined as the difference at the same computation cycles 
then the resulting synchronized error is as given in the second line under Fig. 6.3. Note 
that there is one input cycle for CPU B at which there is no new information available 
using this definition. The resulting error is shown in Fig. 6.4 and, as predicted, stays on 
the same bounding curve, providing a well defined and analytical function. 


n 

CPU-A2 Y2 


n+1 


n+2 
H — 


n 

CPU-Al Yl , 


n+1 

-4- 


n+2 


CPU-B 


Continuous error : y^(n)-Yi(n) Yj(n)-Y,(n+1) Y^(n+1 )-Y,{n+l) 

Synchronized error: "5^(n)-Y,(n) - Y^f n+1 )-Y,( n+1 ) 

Fig. 6.3 Synchronized Error Definition 


Yj(n+2)-Y;(n+2) 
Yj(n+2)-Y,( n+2) 



Fig. 6.4 Synchronized Error at Node A 
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VII. DERIVATION OF MATHEMATICAL MODEL OF ERROR 


Assume that each CPU performs a first order linear differential equation of the form 
X = —aX + bU. Hence the output in the discrete domain at cycle n+1 cam be written as: 


Xn+l — CtXn + fiUn 

where 

a = f(l - e““^) 

and T is the nominal sample interval. 

Let CPU-1 be the standard CPU, the block diagram for CPU-1 is: 


Urc 


ZOH 


aXn + pUn 


^n+l 


where the Zero Order Hold (ZOH) holds the input constant over the sample interval. 


The difference in clock times between CPU-Al and CPU-A2 can be considered as the 
time delay of CPU-A2 with respect to CPU-Al. The block diagram of CPU-A2 then cam 
be drawn as follows: 



where a pure time delay of magnitude r has been added. 

The model error now can be drawn by combining the above diagrams. 



®n+l 


where 


Cn+l — -^n+l ^n+1 

T = {T2-T,jn 


Tl : computational time of CPU-1 
T2 : computational time of CPU-2 
n : computational cycle number 


The mathematical error model now can be derived as follows: 
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Using Laplace Transformation of the elements in the schematic gives: 



The transfer function for CPU-1 is: 


U*{s) s 5 + a 

and the transfer function for CPU-2 is: 

1 >> j. 

U*{s) s s + a 


The transfer function for the error is then: 


(7.1) 


(7.2) 


E(s) _ Xl))!) -A:'(s) _ 1 - e-»7 t 
U*{s) ^*{^) 5 5 -f a 


(7.3) 


Where E(s) is the output resulting from the impulse train input The impulse 

transform may be converted to a Z-transform by 


1) Substituting z = where possible. 

2) Converting the remaining s terms to functions of by 
performing a complex convolution [3] integral of the transfer function 
with the Laplace transform of an impulse train [ i^[5(t)] = 

3) Substituting z = in the resulting equation. 


Applying steps l) and 2) give: 

E(z) ^ / 1 ,, 

U[z) 2'nj z Jc 1 — A(A-|-a) 


(7.4) 
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Using the residue theorem to solve this complex convolution integral gives: 


£(f) ^ ^ V » - 11 „ .> 

U{z) 2 A(A+a) ' ' 

In order to to facilitate evaluating the residues, a partial fraction expansion is applied to 
get: 

E{z) _ 2 - 1 - 1) 

u\z) ~ 2 -^ 1 - 

Evaluating the residues, equation (7.6) becomes: 


o A A -|- <i 


) 


(7.6) 


E{z) _ b z-1 1 - 
U{z) a z 1 — 


(7.7) 


The final result is obtained by substituting z = in (7.7) to get: 


E{z) b z — 1 


U{z) 


a z 


^(1 




(7.9) 


The difference equation for the error is then obtained from the Z-domain transfer function 


as: 

+ -(1 - e-“')(Cf„+i - £/„) (7.9) 

a 

Assuming the CPU’s are initially synchronized, the phase shift, r, beween the CPU’s is a 
time varying function given by: 

r = -AT (7.10) 

where AT is the difference in the clock cycles between CPU-1 and CPU-2. 


At r = T the two CPU’s have become one computation cycle out of phase. In order 
to avoid making the error model dependent on more than one previous error value, the 
error model is redefined at this point to be: 

= (7.11) 

where the superscript ” -f-” denotes the redefined error. For consistency with the 
redefined error, the phase shift, r, is reset to zero: 


r = 0 


(7.12) 
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Both the redefinition of the errors and the resetting of the phase shift to zero for 
consistancy with the redefined errors occur only when the phase shift equals one full 
computational cycle. 

Writing the redefined error at time n ) in terms of the previous error definition 
at time n {E~) gives; 

= = + (7,13) 

EJ = £- + Xi - (7.14) 

The error model is then propagated by equations (7.9) and (7.10) until r= T. At this 
time the error is updated by the increment given in (7.14) and r is reset to zero. Equations 
(7.9) and (7.10) then represent the error between CPU’s 1 and 2 one computational cycle 
apart. Since this corresponds to the two most current values generated by the two CPU’s, 
the error equation continually represents the difference in the two most current CPU 
outputs. The update process must be repeated whenever r = T. 

VIII. ANALYSIS OF ERROR MODEL 
ERROR PREDICTION FOR A-NODE 

The formulas derived in the previous section were implemented and compared with 
errors generated by the simulation program. Predicted error and meatsured error are 
plotted for comparison in Fig. 8.1. 



Fig. 8.1. Error Prediction at A Node 
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The predicted and measured error, based on the synchronized error definition, are seen 
to compare very well. The discontinuity at 2.0 seconds corresponds to the redefinition of 
the error when a full cycle of lag has built up and closely matches the measured value ot 

the redefined error. 


ERROR PREDICTION FOR B-NODE 

In analyzing the error for the A-node CPU’s, the effect of voting logic on the inputs to 
the A-nodes has been neglected because the relatively rapid sensor sampling rate results 
in the three sensor values being very similar. The results are therefore similar regardless of 
the input selected. This is not true for CPU’s at the B and C nodes. There the sampling 
rates are considerably slower so the sampled outputs from the A CPU’s will be significantly 
different as the effective phase lag, , increases. The two types of voting logic to be examined 
are select CPU-1 and Midvalue select. 


Select CPU-1 Logic 


The select CPU-1 logic chooses, as the default, the output of the CPU in the first 
pipeline at the previous node for input to the next node. The error in the B-node CPU^s 
as seen at the C-node CPU’s is shown in Fig. 8.2, where the input to the B-node CPU s 
has been based on the select CPU-1 logic. Fig. 8.3 shows the error, as seen at the C-node, 



Fig. 8.2 Error at B-Node; Select CPU-1 



Fig. 8.3 Synchronized Error at B-Node; Select CPU-1 
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Although the error now is represented by only one of the two bounding curves, the 
function is highly discontinuous. This is due to the low sampling rate applied at the inputs 
to the B-node CPU’s. 

Midvalue Select Logic 

The midvalue select logic chooses, as the default, the output of the CPU at the previous 
node whose value is in the middle of the output range of the CPU’s at that node. The 
error in the B-node CPU’s as seen at the C-node CPU’s is shown in Fig. 8.4, where the 
input to the B-node CPU’s has been based on the Midvalue select logic. Fig. 8.5 shows 
this error using the synchronized error definition. 




Comparing Fig. 8.5 to Fig. 8.3, it can be seen that the Midvalue select produces a 
smoother"error curve than the select CPU-1 logic, so this will be used for the remaining 
analysis. 

Input Smoothing 

The error prediction for A-node CPU’s is good because their inputs are from analog 
devices (sensors,) these inputs are smooth aad continuous. The situation in B (and C) node 
CPU’s is different since the inputs to B node CPU’s, as obtained from the A-node CPU’s, 
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are step functions. Since, the derived math model is bcised on smooth and continuous inputs 
the results will be degraded. 

To resolve the problem of discontinuity, a first order hold wm applied in eaw:h CPU 
to smooth its output. Figure 8.6 shows the meatsured and predicted error with a first 
order hold applied to the output of the B CPU’s. The curve is smooth and, hence a good 
approximation is achieved with the error model. 



rtm 

Fig. 8.6 Error Prediction at B-Node 
ERROR PREDICTION FOR C-NODE 

Figure 8.7 shows the error between the C-node CPU’s. The strange shape is a result of 
two levels of sampling between the continuous input to the A-node CPU’s and the output 
of the C-node CPU’s. Figure 8.8 shows the synchronized error at the C-node and the error 
is seen to stay on the same bounding error curve. The effect of output smoothing at the 
A and B nodes is shown in Fig. 8.9 and compared with the model prediction. The results 
compare quite well. 



Fig. 8.7 Error at C-Node 
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Fig. 8.8 Synchronized Error at C-node 



In Fig. 8.9 the smoothing has been done by interpolation, which assumes that the 
next output is available at the time the smoothed output must be generated. Since this 
is generally not the case, a first order smoothing based on extrapolating the currently 
available data wzis implemented and the results are shown in Fig. 8.10 along with the pre- 
dieted results. The comparison is still quite good and the implementation is now physically 
achievable. 



Fig. 8.10 Error Prediction at C-Node; extrapolation 


17 



SUMMARY OF CPU OPERATIONS 

The following tasks must be performed each cycle by each CPU to allow estimation 
of the error due to asynchronisity given a knowledge of the difference in CPU clock rates 
(AT). 

a. Estimate the error due to asynchronisity from upstreeim CPU’s based on the inputs, 
b. Calculate the output, c. Smooth the output by using a first order hold. 

These three tasks are shown in Fig. 8.11 


Error Estimation 

En+. = - £/„) 

a 

Output Computation 
■^n+l = 

Output smoothing 
X(l)=X„ + (t-nT) 

Fig. 8.11 CPU diagram. 


IX. PARAMETER IDENTIFICATION 

Using the error formula developed in section VII, the difference in CPU clock interval, 
AT, can be identified using the Output Error Method [4]. The identified AT then will 
be used to track the error due to asynchronisity. The mathematical derivation of the 
identification algorithm and the results are presented in this section. 

MATHEMATICAL DERIVATION 

In order to simplify the algorithm, the derivation assumes that the phase shift does not 
exceed one computation cycle. This is insured in practice by resetting the error equation 
whenever the phase reaches a full computation cycle. 

Given a dynamic model of the output difference due to asynchronisity: 
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(9.1) 


E„+i = + -(1 - e-")(!7„+, - U„] 

a 

T = {^)AT = nAT (9.2) 

and given a measurement, E^, of the difference at cycle n, a measurement error may be 
defined as: 

er, = E:C- EniAT) (9.3) 

Where the estimated difference, En{AT), is a function of the difference in clock intervals. 
A cost function is now written in terms of the measurement error as: 

•/(AT) = i^£j(AT) (9.4) 

^ i=0 

Expanding J as a Taylor’s series in AT about an apriori estimate, AT, gives: 


J{AT) « J{AT) + 


lAT-ATI + ij^lAr-ATI^ 


dAT^~~ ' 2d{ATY 

Taking the differential of J due to a change in AT gives: 


(9.5) 


J{AT) - J(AT) = 


dJ d^J — 

+ [AT - 


[dAT d{ATy 


{AT - AT) 


The necessary condition for J to be minimum with respect to AT is = 


gives: 


a.(AT)_a.(Ar)^a^l^^__l^^ 


dAT 


or 


AT = AT - 


d{AT) a(AT)2 
d^J{AT)^~^ 


d{ATY^ 


dJ{AT) 

dAT 


(9.6) 
0. This 

(9.7) 

(9.8) 


Taking the derivative of equation (9.4) with respect to AT at AT gives: 


dJ{AT) 

dAT 




n=0 


derrjAT) 

dAT 


■ d^JjAT) A 


' der,{AT) y 

dAT 


and from 9.3 and 9. 1-9.2 we get: 


8e„(AT) dE„{AT) 

d{AT) d{AT) 


(9.9) 


(9.10) 


(9.11) 
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and 


d(^T) - " aAi^ ^'‘1 


(9.12) 


The complete procedure, based on updating the estimate of AT every N saunples, 
is summarized below. 


1) Propagate error and error sensitivity up to cycle n = N. 

tr, = E^- E„ 

Resynchronization of the computation cycles of the two CPU’s is accomplished by 
resetting the index, n, to zero whenever the phzise shift (nAT) equals one computation 
cycle (T). At this time, the error is also redefined as described in Equation 7.14. The 
equations for propagating the error and error sensitivity are then: 


b 


En+i — e “ En H — (1 — e 
a 


de 


- u„); 

dEn 


dAT 


dAT 


dE, 


_ .-aT ^En , _t. — noATrrr rr 1. 

d(AT)-" [C^n+l-f^n], 

2) Generate first and second gradient of cost function for N cycles 

dJ ^ dCn 

= 2 ^^- 


dAT 


n=0 


dAT 


d{ATY ~ ^^\dAT) 


3) Generate estimate of best AT over N cycles. 


AT = AT- ( 


-1 


dJ 

\d{ATY ) dAT 


Eq = 0. 


dEo 

dAT 


= 0 . 


SUMMARY OF CPU OPERATION 

The total operation required for tracking the difference in CPU outputs due to differ- 
ences in the clock intervals is shown below 
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For previous CPU’s (j = 1,2,3) 

Estimate E rror due to Asynchronisity/ Identify Clock Difference 

dJ 

AT- AT - j 

where 

dJ ^ den 

3AT “ a(AT)2 ~ ^^\dATj 


and 


where 


and 


e„ = 


den _ dEn 
SAT “ SAT 


E„+, = e-'-^En + -(1 - e-")(Cf„+, - Un) 

CL 

T = nAT; 0 < r < T 


a^n+l _ -aT , ^L^-naAT 


d{AT) 


= e 


aAT 


+ n6e-"“^^[Un+i-U„ 


Select Input (j = 1,2,3) 

If e-^ < e U = Midvalue of previous CPU output 
Else Fail CPW 


Compute Output 


Xn+l — «-^n + 0Un 


Smooth output 


X{t) =Xn+ - nr) 
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X. RESULTS OF PARAMETER IDENTIFICATION USING SIMULATION 


The identification algorithm was tested by running the simulation with a specified 
AT between the clocks. The identification algorithm was then applied to the output data, 
initially assuming a AT that was different from the one used in the simulation. The 
algorithm would be considered successful if the estimated AT converged to the actual 
value and the corresponding estimate at the CPU output difference tracked the measured 
value accurately. 

In order to examine the stability of the algorithm, the test was first run with an initial 
estimate of AT that was equal to the actual value. The dynamic model was not updated 
with the estimated AT. Figure 10.1 shows the history of the AT estimate. It is seen 
to initially depart from the correct value but to converge back to the correct value after 
about 35 samples and to hold that value from then on. The corresponding comparison of 
the CPU difference and its estimate is shown in Fig. 10.2 and is seen to be excellent, as 
expected since the correct value of AT is always used. 




22 


The second test was to initiate the identification with an estimate of AT'that waa half 
the actual value. Again the identified value was not used to update the dynamic model. 
The result of the identification is shown in Fig 10.3. Here we see that the correct value has 
again been achieved within 35 samples. Now, however, the estimate degrades significantly 
at the discontinuity associated with the error redefinition. This is due to the fact that the 
estimate is not being fed back to the model to allow the model output to converge to the 
measured CPU difference as can be seen from a comparison of the estimated and actual 
CPU differences in Fig 10.4. 



Fig. 10.3 AT Estimate (AT(0) = |AT*; No Feedback) 



The third test added the updating of the model with the identified AT while using 
the correct initial AT for the parameter estimator. The update was performed every 35 
cycles since this number wzis shown to be adequate to achieve a steady state estimate. 
The results of the identification are shown in Fig 10.5 and the corresponding comparing 
of actual and estimated CPU differences is shown in Fig. 10.6. Now the identification is 
quite good until the discontinuity is reached. After this point, the accuracy degrades. This 
is apparently because the discontinuity generates large transients in the identification zind 
the 35 sample update interval falls in the middle of this transient. 
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In order to avoid this problem, a new criteria for the update was generated. The 
first update occurs automatically after 35 cycles. Successive recursive estimates of AT are 
compared and when the change in these estimates changes sign, it is assumed that the 
result is close to steady state and the model is updated with the currently identified value 
of AT. As a further prcaution, the update is only made if the magnitude of the change is 
below the magnitude of the previous update. The results of this approach are shown in Fig. 
10.7 and 10.8. The identified AT and the output of the model both match the simulation 
values very well. A further improvement was achieved by reducing the magnitude of the 
update by a scale factor to allow a smoother convergence. The update algorithm is then: 


AT= AT- AT= 


1 

1 + W* 


■a2j(Ar)i“^aj(AT) 

. a(AT)2 dAT 


( 10 . 1 ) 


where r is a value less than 1.0 and k is the update number. This scale factor allows the 
amount of the update to be gradually increased from 50% to 100% of the actual estimate 
over a period of time and improves the stability of the algorithm. The output of the model 
corresponding to this modified update algorithm is shown in Fig. 10.9 and is seen to match 
the measured CPU difference quite well. 
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Elinoil AT C-NOOE EIIDOn AT C-NODE 
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actual 

estimated 




As a final test of the algorithm, the previous procedure was run using an initial 
estimate of AT that was 50 % greater than the actual value. The update logic and update 
scale factor were applied as in the previous case and the results are shown in Figures 10.10 
and 10.11. The identification algorithm and model output both track the simulation value 
quite well. 


-4 



Fig. 10.10 AT Estimate (AT(0) = 1.5AT*; Scaled Feedback, Update at Steady State) 



Fig. 10.11 Error Estimate {AT(0) = 1.5AT*; Scaled Feedback, Update at Steady State) 
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XI. SUMMARY OF THE RESULTS OF THIS STUDY 

The following accomplishments were achieved in this study 

a. Definition of Analytic Error Function. The error between two CPU’s was defined 
so as to produce an analytically describable function, 

b. Error Model Derivation. An analytical model was developed to estimate the error 
between CPU’s as a function of the difference in CPU clocks. The error model is based on 
a smooth, continuous input to each CPU; therefore each CPU must perform a first order 
hold to smooth its output. 

c. Accuracy of Error Model Demonstration. The estimated error was shown to be 
within 2% of the measured error when the difference in clock intervals is known, 

d. Identification of Difference in CPU Clock Interval . Since the error formula depends 
only on the difference in CPU clock interval, AT, true AT can be regressively tracked by 
applying Parameter Identification methods. Identified AT was shown to be within 5% 
error of the true AT after 100 cycles. 

e. Updating of Model with identified Clock Intervals . Identified AT was fed back 
into the model and was successfully used to track the error due to asynchronisity, 

XII. CONCLUSIONS 

The ability to model and track the effect of different clock intervals on the relative 
output of redundant CPU’s has been demonstrated using a computer simulation of the 
Redundant Asynchronous Multiprocessor System (RAMPS). This capability effectively 
eliminat the problem of asynchronisity on identifying failures without compromising the 
autonomy of each CPU. The high level of reliability associated with a RAMP system is 
therefore maintained. The identified clock differences could also be used to modify the 
computation algorithms in each CPU to keep the outputs synchronized, again without 
sacrificing their independence. 

XIII. RECOMEND ATIONS FOR FURTHER RESEARCH 

The identification algorithm used in this study was based on the Output Error method. 
The use of Maximum Likelihood identification may allow for more rapid and robust track- 
ing of the clock differences with only a slight increase in computational requirements. The 
use of this technique should therefore be investigated. 

The algorithm modeled in each CPU for this study was a single input/single output 
algorithm. This algorithm was also asymptotically stable. In order to assure generality in 
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the application of this methodology, the use of multi-input multi-output algorithms that 
are not asymptotically stable should also be investigated. 

The use of the identified clock difference to modify the computation algorithm to 
synchronize the CPU’s could keep the error within small bounds without sacrificing CPU 
autonomy. This technique should be investigated. 

The ultimate test of the approach will be to implement it in the RAMPS hardware 
rather than simulate a multiprocessor system on a single CPU. 
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